{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9veUEV0CfmHX"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "id": "BlCInyRifxHS"
      },
      "outputs": [],
      "source": [
        "#@title Copyright 2020 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LRMeRxCfzC4"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/boundless\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/boundless.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/hub/blob/master/examples/colab/boundless.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/hub/examples/colab/boundless.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://tfhub.dev/s?q=google%2Fboundless\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" /\u003eSee TF Hub models\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOjczJJ4gWHS"
      },
      "source": [
        "# Boundless Colab\n",
        "\n",
        "Welcome to the Boundless model Colab! This notebook will take you through the steps of running the model on images and visualize the results.\n",
        "\n",
        "## Overview\n",
        "\n",
        "Boundless is a model for image extrapolation. This model takes an image, internally masks a portioin of it ([1/2](https://tfhub.dev/google/boundless/half/1), [1/4](https://tfhub.dev/google/boundless/quarter/1), [3/4](https://tfhub.dev/google/boundless/three_quarter/1)) and completes the masked part. For more details refer to [Boundless: Generative Adversarial Networks for Image Extension](https://arxiv.org/pdf/1908.07007.pdf) or the model documentation on TensorFlow Hub."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hDKbpAEZf8Lt"
      },
      "source": [
        "## Imports and Setup\n",
        "\n",
        "Lets start with the base imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xJMFtTqPr7lf"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from io import BytesIO\n",
        "from PIL import Image as PilImage\n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "from six.moves.urllib.request import urlopen"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pigUDIXtciQO"
      },
      "source": [
        "## Reading image for input\n",
        "\n",
        "Lets create a util method to help load the image and format it for the model (257x257x3). This method will also crop the image to a square to avoid distortion and you can use with local images or from the internet."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KTEVPgXH6rtV"
      },
      "outputs": [],
      "source": [
        "  def read_image(filename):\n",
        "    fd = None\n",
        "    if(filename.startswith('http')):\n",
        "      fd = urlopen(filename)\n",
        "    else:\n",
        "      fd = tf.io.gfile.GFile(filename, 'rb')\n",
        "\n",
        "    pil_image = PilImage.open(fd)\n",
        "    width, height = pil_image.size\n",
        "    # crop to make the image square\n",
        "    pil_image = pil_image.crop((0, 0, height, height))\n",
        "    pil_image = pil_image.resize((257,257),PilImage.ANTIALIAS)\n",
        "    image_unscaled = np.array(pil_image)\n",
        "    image_np = np.expand_dims(\n",
        "        image_unscaled.astype(np.float32) / 255., axis=0)\n",
        "    return image_np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lonrLxuKcsL0"
      },
      "source": [
        "## Visualization method\n",
        "\n",
        "We will also create a visuzalization method to show the original image side by side with the masked version and the \"filled\" version, both generated by the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j7AkoMFG7r-O"
      },
      "outputs": [],
      "source": [
        "def visualize_output_comparison(img_original, img_masked, img_filled):\n",
        "  plt.figure(figsize=(24,12))\n",
        "  plt.subplot(131)\n",
        "  plt.imshow((np.squeeze(img_original)))\n",
        "  plt.title(\"Original\", fontsize=24)\n",
        "  plt.axis('off')\n",
        "  plt.subplot(132)\n",
        "  plt.imshow((np.squeeze(img_masked)))\n",
        "  plt.title(\"Masked\", fontsize=24)\n",
        "  plt.axis('off')\n",
        "  plt.subplot(133)\n",
        "  plt.imshow((np.squeeze(img_filled)))\n",
        "  plt.title(\"Generated\", fontsize=24)\n",
        "  plt.axis('off')\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8rwaCWmxdJGH"
      },
      "source": [
        "## Loading an Image\n",
        "\n",
        "We will load a sample image but fell free to upload your own image to the colab and try with it. Remeber that the model have some limitations regarding human images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "92w-Jfbm60XA"
      },
      "outputs": [],
      "source": [
        "wikimedia = \"https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/Nusfjord_road%2C_2010_09.jpg/800px-Nusfjord_road%2C_2010_09.jpg\"\n",
        "# wikimedia = \"https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/Beech_forest_M%C3%A1tra_in_winter.jpg/640px-Beech_forest_M%C3%A1tra_in_winter.jpg\"\n",
        "# wikimedia = \"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Marmolada_Sunset.jpg/640px-Marmolada_Sunset.jpg\"\n",
        "# wikimedia = \"https://upload.wikimedia.org/wikipedia/commons/thumb/9/9d/Aegina_sunset.jpg/640px-Aegina_sunset.jpg\"\n",
        "\n",
        "input_img = read_image(wikimedia)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4lIkmZL_dtyX"
      },
      "source": [
        "## Selecting a model from TensorFlow Hub\n",
        "\n",
        "On TensorFlow Hub we have 3 versions of the Boundless model: Half, Quarter and Three Quarters.\n",
        "In the following cell you can chose any of them and try on your image. If you want to try with another one, just chose it and execture the following cells."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B3myNctEQ5GE"
      },
      "outputs": [],
      "source": [
        "#@title Model Selection { display-mode: \"form\" }\n",
        "model_name = 'Boundless Quarter' # @param ['Boundless Half', 'Boundless Quarter', 'Boundless Three Quarters']\n",
        "model_handle_map = {\n",
        "    'Boundless Half' : 'https://tfhub.dev/google/boundless/half/1',\n",
        "    'Boundless Quarter' : 'https://tfhub.dev/google/boundless/quarter/1', \n",
        "    'Boundless Three Quarters' : 'https://tfhub.dev/google/boundless/three_quarter/1'\n",
        "}\n",
        "\n",
        "model_handle = model_handle_map[model_name]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aSJFeNNSeOn8"
      },
      "source": [
        "Now that we've chosen the model we want, lets load it from TensorFlow Hub.\n",
        "\n",
        "**Note**: You can point your browser to the model handle to read the model's documentation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0IDKMNyYSWsj"
      },
      "outputs": [],
      "source": [
        "print(\"Loading model {} ({})\".format(model_name, model_handle))\n",
        "model = hub.load(model_handle)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4G7CPOaeuQb"
      },
      "source": [
        "## Doing Inference\n",
        "\n",
        "The boundless model have two ouputs:\n",
        "\n",
        "*   The input image with a mask applied\n",
        "*   The masked image with the extrapolation to complete it\n",
        "\n",
        "we can use these two images to show a comparisson visualization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W7uCAuKxSd-M"
      },
      "outputs": [],
      "source": [
        "result = model.signatures['default'](tf.constant(input_img))\n",
        "generated_image =  result['default']\n",
        "masked_image = result['masked_image']\n",
        "\n",
        "visualize_output_comparison(input_img, masked_image, generated_image)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "boundless.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
